People can utilize a camera of a device to capture an image of an object. For example, a user can use a camera to capture a still image or video of things in the environment in the camera's field of view. There might be multiple objects in the captured image or video. In some instances, the multiple objects may need to be tracked as the device moves to be able to provide various features. For example, the multiple objects may need to be tracked so that augmented reality techniques can be accurately applied to the captured image or video to insert virtual objects. However, tracking multiple objects using the device can put a strain on computing resources and on device's battery life.